Repository navigation
Native: opt-in capabilities digest instead of ~17 KB server_capabilities on every worker answer - #1262
Conversation
Native sent every answer uncompressed and advertised response_compression: []. PHP Server gzip- or deflate-encodes JSON answers of 1024 bytes or more when Accept-Encoding allows it (DW_COMPRESSION_ENABLED, default true). On a Dev cell behind a Cloudflare tunnel the origin answers are the cell's egress, which is shaped to 4 Mbit/s. The Cloudflare edge always asks the origin for "br, gzip" and decodes for clients that did not ask, so this reduces tunnel egress without any SDK change. Lab measurement (DW Standard Workflow v1, 1 wf/s, 4 workers): 384.8 KB -> 106.0 KB of origin response bytes per workflow. - New response_compression middleware, outermost on the router: PHP's encoding preference (gzip over deflate at equal q, *, q=0), JSON content types only, 1024-byte minimum, existing Content-Encoding left alone, Vary: Accept-Encoding. Only in-memory (exact-sized) bodies are encoded; streamed answers pass through. - DW_COMPRESSION_ENABLED (and its legacy name) is honored instead of ignored. - cluster info advertises ["gzip", "deflate"] when enabled, as PHP does; the native deviation is removed from the parity fixture. - History-page gzip reuses the same RFC 1952 encoder. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
SDK side: durable-workflow/sdk-php#446. Compression: #1261. |
Review fixes: - Bodies above 256 KB are encoded on a blocking thread (spawn_blocking). At roughly 30 MB/s for level 6, an 8 MB body would otherwise hold an async worker for about 300 ms on a 2-thread runtime. - HTTP's deflate coding is the zlib format (RFC 1950), not raw DEFLATE; deflate answers are now zlib-wrapped. - Tests: a large answer encodes on the blocking path and round-trips for gzip and deflate, the zlib header is checked, and an ignored benchmark reports encode cost by size. Co-Authored-By: Claude Opus 5.5 <[email protected]>
encode_cost (release, lab): 14 KB in 0.6 ms, 217 KB in 10 ms, 7 MB in 196 ms (22-36 MB/s). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every worker-plane JSON answer carries the ~17 KB server_capabilities object, and a completion that carries a poll carries a second copy in its answer. For DW Standard Workflow v1 that is about 18 copies (~320 KB) of the ~385 KB of responses per workflow, and it dominated a Dev cell's 4 Mbit/s egress. A worker that sends X-Durable-Workflow-Server-Capabilities-Digest opts in. When the header names the current digest, each copy of the current capabilities (top level, or inside a carried poll's answer) is replaced by the digest string; otherwise each copy is sent in full with a digest member for the worker to keep. Requests without the header are unchanged, so existing SDKs see the same bytes. Contract: docs/contracts/server-capabilities-digest.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fixes: - The contract states that the server replaces copies only with the digest that request sent, so a worker resolves a digest string against the digest and object it captured when sending that request. A worker that kept only its latest digest could reject a committed answer during a rolling deploy. Version names and PR links are removed from the contract prose. - A router-level test sends Accept-Encoding: gzip and the digest header and asserts the digest is applied inside a gzip-encoded body (compression is the outer layer). - The header is documented in server-reference.md and rust-server-port.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
7087d39 to
93be253
Compare
|
Review fixes pushed. Rebased on #1261 (merge #1261 first); compression stays the outermost layer. Added the router-level gzip+digest test, the per-request resolution rule in the contract (version names and PR links removed), and documentation in server-reference.md and rust-server-port.md. Local runs: Draft 3.17 release-notes items: Smaller worker and client responsesNative now sends far fewer response bytes. In a lab run of the standard
|
The elided empty poll is under the 1024-byte compression threshold, so the second answer is now a leased workflow task: still over the threshold with its capabilities elided, so both layers apply to it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|



Why
Every worker-plane JSON answer carries
server_capabilities, as PHP'sWorkerProtocol::jsondoes (server#1001). A completion that carries a poll carries a second copy inside itsanswer. On native that object is about 17 KB.For DW Standard Workflow v1 (one start, one activity, one completion), a worker receives about 18 copies per workflow, roughly 320 KB of the ~385 KB of responses. That traffic was what saturated the Cloud Dev cell's 4 Mbit/s egress at 1.5 wf/s. In the 2 wf/s step, the saturation also cost a workflow its activity lease (sdk-php#445).
Per-endpoint response bytes per workflow (lab,
main, 1 wf/s, 4 workers, 1 s task polls):server_capabilitiesPOST worker/query-tasks/poll(empty)POST worker/workflow-tasks/pollPOST worker/activity-tasks/poll(always empty: activities arrive carried)POST worker/workflow-tasks/{id}/complete(+ carried activity poll)POST worker/activity-tasks/{id}/complete(+ carried workflow poll)POST worker/heartbeatWhat
This adds an opt-in, backward-compatible digest. The contract is in
docs/contracts/server-capabilities-digest.md.X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none.activity_poll.answerandworkflow_poll.answer.digestmember added.The SDK side is in durable-workflow/sdk-php (linked below). Its transport restores the held object before any caller sees the answer, covering success bodies, refusals and poll frames.
Measured (lab)
Setup
main, in a 1 vCPU / 1 GiB Dev-like slice.br, gzip, and logs origin bytes.mainmain+ #1261 (compression)main+ this PR, compression offmain+ #1261 + this PRCPU (lab slice, 1 vCPU, including MySQL):
main90%; Native: compress JSON responses like PHP's CompressResponse (Dev egress 385 → 106 KB/workflow) #1261 + this PR + SDK 92%.p50 and p95 latency were unchanged (1.04 s / 1.09 s).
Tests
capabilities_digest):noneor a stale digest, the copies are tagged, and the digest covers the untagged object.tests/execution.rs::the_capabilities_digest_is_applied_inside_gzip_compression): a request withAccept-Encoding: gzipand the digest header gets a gzip body whose decodedserver_capabilitiesis the digest string.cargo test --lib -- capabilities_digest runtime::http workflow_task_guard carried_polls: 16 passed.cargo test --test execution -- carried: 7 passed.🤖 Generated with Claude Code